基于POMDP强化学习的动态频谱分配算法

doi:10.13190/jbupt.200906.125.tangl

北京邮电大学学报 ›› 2009, Vol. 32 ›› Issue (6): 125-129.doi: 10.13190/jbupt.200906.125.tangl

基于POMDP强化学习的动态频谱分配算法

唐伦;陈前斌;曾孝平

（1.重庆大学通信工程学院，重庆 400044；
2.重庆邮电大学通信与信息工程学院，重庆 400065）

收稿日期:2008-10-05 修回日期:2009-08-31 出版日期:2009-12-28 发布日期:2009-12-28
通讯作者: 唐伦

A Novel Dynamic Spectrum Allocation Algorithm Based on POMDP Reinforcement Learning

TANG Lun;CHEN Qian-bin;ZENG Xiao-ping

(1.College of Communication Engineering, Chongqing University, Chongqing 400044, China;
2.School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China)

Received:2008-10-05 Revised:2009-08-31 Online:2009-12-28 Published:2009-12-28
Contact: TANG Lun

摘要/Abstract

摘要：

提出基于VCG机制的动态频谱分配博弈模型，解决了认知无线网络环境存在的信息约束限制、分布式特性和频谱分配动态、复杂性问题；提出一种基于动态频谱分配的部分可观察马尔可夫决策过程（POMDP）强化学习算法. 认知用户通过对历史信息的观察、统计，为提高竞拍策略的奖赏值而进行不断的学习获取最优竞拍策略. 将POMDP强化学习转变为信度状态马尔可夫决策过程 (belief MDP)最优策略学习. 采用值迭代算法求解信度状态MDP模型的解. 仿真结果表明，基于POMDP强化学习算法可显著改善认知用户的行为，提高动态频谱分配性能.

关键词: 动态频谱分配, VCG机制, 认知网络

Abstract:

A game model based on Vickrey-Clarke-Groves (VCG) mechanism for dynamic spectrum allocation is presented, to solve the complexity problem of the dynamic spectrum allocation and reduce information exchange during the dynamic spectrum allocation. Further, a partially observable Markov decision processes (POMDP) reinforcement learning algorithm is presented. Through the observation and statistics of historical information, the secondary users enhance the reward value of bidding strategy by continuous learning, so as to obtain the optimal bidding strategy
. Finally, the POMDP reinforcement learning algorithm is transformed into optimal strategy learning algorithm of belief Markov decision processes(MDP), which is solved by using the value iteration algorithm. The simulation results reveales that the POMDP reinforcement learning algorithm can significantly improve the performance of dynamic spectrum allocation.

Key words: dynamic spectrum allocation, Vickrey-Clarke-Groves mechanism, cognitive network

唐伦;陈前斌;曾孝平. 基于POMDP强化学习的动态频谱分配算法[J]. 北京邮电大学学报, 2009, 32(6): 125-129.

TANG Lun;CHEN Qian-bin;ZENG Xiao-ping. A Novel Dynamic Spectrum Allocation Algorithm Based on POMDP Reinforcement Learning[J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2009, 32(6): 125-129.

参考文献

[1] Mitola J, Maguire G. Cognitive radio: making software radios more personal[J]. IEEE Personal Commun, 1999, 6(4): 13-18.
[2] 曾志民, 刘琦, 冯春燕, 等. 认知无线电网络自适应带宽分配机制及性能分析[J]. 北京邮电大学学报, 2009, 32(1): 42. Zeng Zhimin, Liu Qi, Feng Chunyan, et al. An adaptive bandwidth allocation mechanism and performance analysis in cognitive radio networks[J]. Journal of Beijing University of Posts and Telecommunications, 2009, 32(1): 42.
[3] 邱晶, 周正. 认知无线电网络中的分布式动态频谱共享[J]. 北京邮电大学学报, 2009, 32(1): 72. Qiu Jing, Zhou Zheng. Dynamic spectrum sharing strategy in cognitive radio systems[J]. Journal of Beijing University of Posts and Telecommunications, 2009, 32(1): 72.
[4] Nie N, Comaniciu C. Adaptive channel allocation spectrum etiquette for cognitive radio networks//Proceedings of IEEE DySPAN'05. Baltimore: , 2005: 269-278.
[5] Niyato D, Hossain E. Competitive spectrum sharing in cognitive radio networks: a dynamic game approach[J]. IEEE Transactions on Wireless Communications, 2008, 7(7): 2651-2660.
[6] Zhu Ji, Liu K J R. Dynamic spectrum sharing: a game theoretical overview[J]. Communications Magazine, 2007, 45(5): 88-94.
[7] Chou C T, Shankar N S, Kim H, et al. What and how much to gain by spectrum agility?[J]. IEEE Journal Sel Areas Commun, 2007, 25(3): 576-588.
[8] Wang Beibei, Zhu Ji. Self-learning repeated game framework for distributed primary-prioritized dynamic spectrum access//Proceedings of IEEE SECON 2007. San Diego:, 2007: 631-638.
[9] Nisan N. Algorithmic game theory[M]. Cambridge: Cambridge University Press, 2007.
[10] Cao L, Zheng H. Spectrum allocation in Ad hoc networks via local bargaining//IEEE SECON. Santa Clara: , 2005: 475-486.
[11] IEEE 802.11e/D5.0—2003, Wireless medium access control (MAC) and physical layer (PHY) specifications: medium access control (MAC) enhancements for quality of service (QoS)[S]. 2003.
[12] Klemperer P. Auction theory: a guide to the literature[J]. Journal Economics Surveys, 1999, 13(3): 227-286.
[13] Kloeck C, Jaekel H, Jondral F. Auction sequence as a new resource allocation mechanism//Proceedings of VTC'05. Dallas: , 2005: 240-244.
[14] Boyd S P, Vandenberghe L. Convex optimization[M]. Cambridge: Cambridge University Press, 2004.
[15] Kaelbling L P, Littman M L, Cassandra A R. Planning and acting in partially observable stochastic domains[J]. Artificial Intelligence, 1998, 10(1): 99-134.

基于POMDP强化学习的动态频谱分配算法

A Novel Dynamic Spectrum Allocation Algorithm Based on POMDP Reinforcement Learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 3

编辑推荐

Metrics

本文评价

[1]	陈雪艳, 郭莉, 林家儒. 认知双向中继网络下分布式鲁棒波束成形[J]. 北京邮电大学学报, 2014, 37(6): 91-94.
[2]	陈亮纪红李屹李曦. 认知无线电网络中基于演化博弈的协作频谱感知技术[J]. 北京邮电大学学报, 2012, 35(1): 28-31.
[3]	王景尧,宋梅,张勇，张英海. 认知网络中基于小波分析的切换判决[J]. 北京邮电大学学报, 2010, 33(6): 98-102.